Projector-camera misalignment correction for structured light systems

ABSTRACT

A method of misalignment correction in a structured light device is provided that includes extracting features from a first captured image of a scene, wherein the first captured image is captured by an imaging sensor component of the structured light device, and wherein the first captured image includes a pattern projected into the scene by a projector component of the structured light device, matching the features of the first captured image to predetermined features of a pattern image corresponding to the projected pattern to generate a dataset of matching features, determining values of alignment correction parameters of an image alignment transformation model using the dataset of matching features, and applying the image alignment transformation model to a second captured image using the determined alignment correction parameter values.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Patent ApplicationSer. No. 61/806,117, filed Mar. 28, 2013, which is incorporated byreference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to correction ofprojector-camera misalignment in structured light systems.

2. Description of the Related Art

In structured light imaging systems, a projector-camera pair is used toestimate the three-dimensional (3D) depth of a scene and shape ofobjects in the scene. The principle behind structured light imaging isto project patterns on objects/scenes of interest and capture imageswith the projected pattern. The depth is estimated based on variationsof the captured pattern in comparison to the projected pattern. In suchimaging systems, the relative position of the camera with respect to theprojector is typically fixed and the camera-projector pair is initiallycalibrated, e.g., by the maker of the system or by a user followingcalibration instructions.

After physical calibration, a further step involving projecting achecker board pattern on a plane surface at different depths may berequired. Note that this calibration may require that the projector hasthe capability to project more than one pattern. Calibration isperformed by finding correspondences between locations of corners of thecheckerboard pattern in captured and projected images. Such calibrationmay require manual intervention in which the plane surface is moved bythe user in front of the camera-projector pair. This calibration isgenerally performed once as the assumption is that there will be nochange in the relative position of the camera-projector pair.

Such calibration is generally sufficient as long as the camera-projectorpair remains stationary. However, in practical applications, astructured light system may not be stationary. For example, a structuredlight system may be mounted on an unstable platform or used in ahandheld device. In such applications, the relative position of thecamera and projector can be altered unintentionally over time or due tomanufacturing imperfections and environmental factors, thusnecessitating re-calibration.

SUMMARY

Embodiments of the present invention relate to methods, apparatus, andcomputer readable media for correction of projector-camera misalignmentin structured light systems. In one aspect, a method of misalignmentcorrection in a structured light device is provided that includesextracting features from a first captured image of a scene, wherein thefirst captured image is captured by an imaging sensor component of thestructured light device, and wherein the first captured image includes apattern projected into the scene by a projector component of thestructured light device, matching the features of the first capturedimage to predetermined features of a pattern image corresponding to theprojected pattern to generate a dataset of matching features,determining values of alignment correction parameters of an imagealignment transformation model using the dataset of matching features,and applying the image alignment transformation model to a secondcaptured image using the determined alignment correction parametervalues.

In one aspect, structured light device is provided that includes animaging sensor component configured to capture images of a scene, aprojector component configured to project a pattern into the scene, amemory configured to store predetermined features of a pattern imagecorresponding to the pattern, means for extracting features from a firstimage of the scene captured by the imaging sensor component, wherein thefirst image includes the pattern projected into the scene by theprojector component, means for matching the features of the first imageto the predetermined features to generate a dataset of matchingfeatures, means for determining values of alignment correctionparameters of an image alignment transformation model using the datasetof matching features, and means for applying the image alignmenttransformation model with the determined alignment correction parametervalues to a second image of the scene captured by the imaging sensorcomponent.

In one aspect, a non-transitory computer-readable medium storinginstructions that, when executed by at least one processor in astructured light device, cause a method of misalignment correction to beperformed. The method includes extracting features from a first capturedimage of a scene, wherein the first captured image is captured by animaging sensor component of the structured light device, and wherein thefirst captured image includes a pattern projected into the scene by aprojector component of the structured light device, matching thefeatures of the first captured image to predetermined features of apattern image corresponding to the projected pattern to generate adataset of matching features, determining values of alignment correctionparameters of an image alignment transformation model using the datasetof matching features, and applying the image alignment transformationmodel to a second captured image using the determined alignmentcorrection parameter values.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments in accordance with the invention will now bedescribed, by way of example only, and with reference to theaccompanying drawings:

FIG. 1 is a block diagram of an example digital structured light device;

FIG. 2 is a flow diagram of a method; and

FIG. 3 is an example.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

As previously mentioned, in some applications, the camera-projector pairin a structured light imaging system may become misaligned during use,thus necessitating re-calibration. The misalignments may include, forexample, translation, rotation, scaling, and/or horizontal/verticalskew. It may not be practical to perform the prior art calibrationtechniques previously described. Moreover, some calibration techniquesare not suited to systems in which the projector can project just onefixed projection pattern.

Embodiments of the invention provide for automatic calibration of acamera-projector pair in a structured light imaging system as theimaging system is used. Unlike prior art techniques, the providedautomatic calibration does not require a change in the projected patternor a flat surface in front of the camera-projector pair.

FIG. 1 is a block diagram of an example digital structured light device100. The digital structured light device 100 includes a structured lightimaging system 102, an image and depth processing component 104, a videoencoder component 118, a memory component 110, a video analyticscomponent 112, a camera controller 114, and a network interface 116. Thecomponents of the camera 100 may be implemented in any suitablecombination of software, firmware, and hardware, such as, for example,one or more digital signal processors (DSPs), microprocessors, discretelogic, application specific integrated circuits (ASICs),field-programmable gate arrays (FPGAs), etc. Further, softwareinstructions may be stored in memory in the memory component 110 andexecuted by one or more processors.

The structured light imaging system 102 includes an imaging sensorcomponent 106, a projector component 108, and a controller component 109for capturing images of a scene. The imaging sensor component 106 is animaging sensor system arranged to capture image signals of a scene andthe projector component 108 is a projection system arranged to project apattern of light into the scene. The imaging sensor component 106includes a lens assembly, a lens actuator, an aperture, and an imagingsensor. The projector component 108 includes a projection lens assembly,a lens actuator, an aperture, a light source, and projection circuitry.The structured light imaging system 102 also includes circuitry forcontrolling various aspects of the operation of the system, such as, forexample, aperture opening amount, exposure time, synchronization of theimaging sensor component 106 and the projector component 108, etc. Thecontroller component 109 includes functionality to convey controlinformation from the camera controller 114 to the imaging sensorcomponent 106, the projector component 108, to convert analog imagesignals from the imaging sensor component 106 to digital image signals,and to provide the digital image signals to the image and depthprocessing component 104.

In some embodiments, the imaging sensor component 106 and the projectioncomponent 108 may be arranged vertically such that one component is ontop of the other, i.e., the two components have a vertical separationbaseline. In some embodiments, the imaging sensor component 106 and theprojection component 108 may be arranged horizontally such that onecomponent is next to the other, i.e., the two components have ahorizontal separation baseline.

The image and depth processing component 104 divides the incomingdigital signal(s) into frames of pixels and processes each frame toenhance the image data in the frame. The processing performed mayinclude one or more image enhancement techniques such as, for example,one or more of black clamping, fault pixel correction, color filterarray (CFA) interpolation, gamma correction, white balancing, colorspace conversion, edge enhancement, denoising, contrast enhancement,detection of the quality of the lens focus for auto focusing, anddetection of average scene brightness for auto exposure adjustment oneach of the left and right images. In addition, the image and depthprocessing component 104 may perform an embodiment of the method forprojector-camera misalignment correction of FIG. 2 to enhance the imagedata, i.e., to adjust the image data for any misalignment between theimaging component 106 and the projection system 108. The misalignmentcorrection may be performed before or after the other image enhancementtechniques.

With either a horizontal or vertical component baseline, the field ofview (FOV) of the imaging sensor component 106 may be larger than thatof the projector component 108. The projected pattern varies in thecaptured image along the direction (epipolar lines) of the imagingsensor-projector separation based on the depth of objects in a scene.Thus, a wider FOV is needed to capture the projected patternirrespective of the depth of objects in the scene. Accordingly, theimage and depth processing component 104 may perform rectification oneach captured image to correct for the FOV variation in the directionperpendicular to the component baseline. Among other operations, therectification processing may include discarding any portions of thecaptured image that are outside the boundaries of the projected pattern.The rectification parameters may be determined during offlinecalibration of the imaging sensor-projector pair and stored in thememory component 110 for use by the image and depth processing component104. In some embodiments, the rectification parameters may be updated asneeded during operation of the digital structured light device 100.

The image and depth processing component 104 then uses the enhancedimage data to generate a depth image. Any suitable algorithm may be usedto generate the depth image from the enhanced image data. The enhancedcaptured image is provided to the video encoder component 108 and depthimage and enhanced captured image are provided to the video analyticscomponent 112.

The video encoder component 108 encodes the image in accordance with avideo compression standard such as, for example, the Moving PictureExperts Group (MPEG) video compression standards, e.g., MPEG-1, MPEG-2,and MPEG-4, the ITU-T video compressions standards, e.g., H.263 andH.264, the Society of Motion Picture and Television Engineers (SMPTE)421 M video CODEC standard (commonly referred to as “VC-1”), the videocompression standard defined by the Audio Video Coding StandardWorkgroup of China (commonly referred to as “AVS”), the ITU-T/ISO HighEfficiency Video Coding (HEVC) standard, etc.

The memory component 110 may be on-chip memory, external memory, or acombination thereof. Any suitable memory design may be used. Forexample, the memory component 110 may include static random accessmemory (SRAM), dynamic random access memory (DRAM), synchronous DRAM(SDRAM), read-only memory (ROM), flash memory, a combination thereof, orthe like. Various components in the digital structured light device 100may store information in memory in the memory component 110 as a videostream is processed. For example, the video encoder component 108 maystore reference data in a memory of the memory component 110 for use inencoding frames in the video stream. The memory component 110 may alsostore a pattern image and associated features for use by the image anddepth processing component 104 in performing the method of FIG. 2.Pattern images and associated features are explained in more detail inreference to FIG. 2.

Further, the memory component 110 may store any software instructionsthat are executed by one or more processors (not shown) to perform someor all of the described functionality of the various components. Some orall of the software instructions may be initially stored in acomputer-readable medium such as a compact disc (CD), a diskette, atape, a file, memory, or any other computer readable storage device andloaded and stored on the digital structured light device 100. In somecases, the software instructions may also be sold in a computer programproduct, which includes the computer-readable medium and packagingmaterials for the computer-readable medium. In some cases, the softwareinstructions may be distributed to the digital structured light device100 via removable computer readable media (e.g., floppy disk, opticaldisk, flash memory, USB key), via a transmission path from computerreadable media on another computer system (e.g., a server), etc.

The camera controller component 114 controls the overall functioning ofthe digital structured light device 100. For example, the cameracontroller component 114 may adjust the focus and/or exposure of thestructured light imaging system 102 based on the focus quality and scenebrightness, respectively, determined by the image and depth processingcomponent 104. The camera controller component 114 also controls thetransmission of the encoded video stream via the network interfacecomponent 116 and may control reception and response to camera controlinformation received via the network interface component 116.

The network interface component 116 allows the digital structured lightdevice 100 to communicate with other systems, e.g., a monitoring system,via a network such as, for example, a local area network (LAN), a widearea network (WAN) such as the Internet, a cellular network, any othersimilar type of network and/or any combination thereof. The networkinterface component 116 may use any suitable network protocol(s).

The network interface component 524 may provide an interface for a wiredlink, such as an Ethernet cable or the like, and/or a wireless link via,for example, a local area network (LAN), a wide area network (WAN) suchas the Internet, a cellular network, any other similar type of networkand/or any combination thereof.

The video analytics component 112 analyzes the content of images in thecaptured video stream to detect and determine temporal events not basedon a single image. The analysis capabilities of the video analyticscomponent 112 may vary in embodiments depending on such factors as theprocessing capability of the digital structured light device 100, theparticular application for which the digital structured light device isbeing used, etc. For example, the analysis capabilities may range fromvideo motion detection in which motion is detected with respect to afixed background model to people counting, detection of objects crossinglines or areas of interest, vehicle license plate recognition, objecttracking, facial recognition, automatically analyzing and taggingsuspicious objects in a scene, activating alarms or taking other actionsto alert security personnel, etc.

FIG. 2 is a flow diagram of a method for projector-camera (imagingsensor) misalignment correction that may be performed, for example, bythe digital structured light device 100 of FIG. 1. Initially, featuresare extracted 200 from a captured image of a scene. The captured imageincludes both the scene and a pattern projected into the scene. Anysuitable feature detection algorithm may be used. Examples of suitablealgorithms include SIFT (Scale Invariant Feature Transform) and Harriscorner detection. The SIFT algorithm is described in D. G. Lowe,“Distinctive Image Features from Scale-Invariant Keypoints,”International Journal of Computer Vision, Vol. 60, No. 2, pp. 91-110,November 2004 (“Lowe” herein). Harris corner detection is described inC. Harris and M. Stephens, “A Combined Corner and Edge Detector,”Proceedings of Fourth Alvey Vision Conference, pp. 147-151, 1988. Inanother example, if the projected pattern is a simple stripe pattern asis the case with time-multiplexed patterns, a search based algorithm canbe used to detect the corners of these stripes to extract the features.

Next, matches between the extracted features of the captured image andfeatures of the pattern image are found 202. The pattern image is animage of the pattern that was projected into the scene when the capturedimage was captured. The pattern image and features for the pattern imageare predetermined and are stored in the digital structured light system.The features for the pattern image are extracted using the same featureextraction algorithm used to extract features from the captured image.The predetermined pattern image and the associated features are storedin the digital structured light system. The pattern image may be anysuitable two-dimensional (2D) high frequency binary pattern. Binarypatterns with high-frequency details contain several distinctfeatures/corners which can be detected by a suitable feature detectionalgorithm. Binary patterns are preferred over continuous patternsbecause continuous patterns do not have distinct features that can beused for feature matching. In some embodiments, the pattern image is afixed, binary pattern image of pseudo-random dots.

Any suitable algorithm may be used to find matches between the featuresof the captured image and the features of the pattern image. Examples ofsuitable matching algorithms include the matching algorithm described inLowe and the BRIEF (Binary Robust Independent Elementary Features)algorithm described in M. Calonder et al., “BRIEF: Binary RobustIndependent Elementary Features,” Proceedings of the 11^(th) EuropeanConference on Computer Vision Part IV, pp. 778-792, September 2010.

The matched features are then used to determine 204 alignment correctionparameters of a 2D image alignment transformation model. In essence, thematching features are used to model the relationship between thecaptured image and the pattern image in terms of parameters of the 2Dimage alignment transformation model used. In some embodiments, anaffine transformation model is used. The classic affine transformationmodel is given as follows:

$\begin{bmatrix}x^{\prime} \\y^{\prime}\end{bmatrix} = {{\begin{bmatrix}{1 + c} & {- s} \\s & {1 + c}\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}} + \begin{bmatrix}d_{x} \\d_{y}\end{bmatrix}}$where d_(x) is the horizontal translation parameter, d_(y) is thevertical translation parameter, s is the rotation parameter, and c isthe scaling parameter. This transformation model is modified dependingupon the separation baseline of the camera and the imaging system, i.e.,the directional component along the baseline is ignored because thedifference observed in the baseline direction includes disparity inaddition to misalignment, whereas the non-baseline direction includesjust the misalignment. If the separation baseline is horizontal, thehorizontal component of the model is removed, resulting in the followingmodel with three parameters:

$\left\lbrack y^{\prime} \right\rbrack = {{\begin{bmatrix}s & {1 + c}\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}} + {\left\lbrack d_{y} \right\rbrack.}}$If the separation baseline is vertical, the vertical component of themodel is removed, resulting in the following model with threeparameters:

$\left\lbrack x^{\prime} \right\rbrack = {{\begin{bmatrix}{1 + c} & {- s}\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}} + {\left\lbrack d_{x} \right\rbrack.}}$

In some embodiments, a perspective transformation model, also referredto as a perspective transformation model or homography, is used. Thismodel uses the same parameters as the affine model along with twoadditional parameters. This transformation model can be written as

$\begin{bmatrix}X \\Y \\Z\end{bmatrix} = {\begin{bmatrix}{1 + c} & {- s} & d_{x} \\s & {1 + c} & d_{y} \\g & h & 1\end{bmatrix}\begin{bmatrix}x \\y \\1\end{bmatrix}}$where g and h are parameters that enable perspective warp. Cartesiancoordinates can be calculated as x′=X/Z and y′=Y/Z. Similar to theaffine model, this transformation model is modified depending upon theseparation baseline of the camera and the imaging system. If theseparation baseline is horizontal, the horizontal component of the modelis removed, resulting in the following model with five parameters:

$\begin{bmatrix}Y \\Z\end{bmatrix} = {{\begin{bmatrix}s & {1 + c} & d_{y} \\g & h & 1\end{bmatrix}\begin{bmatrix}x \\y \\1\end{bmatrix}}.}$If the separation baseline is vertical, the vertical component of themodel is removed, resulting in the following model with five parameters:

$\begin{bmatrix}X \\Z\end{bmatrix} = {{\begin{bmatrix}{1 + c} & {- s} & d_{x} \\g & h & 1\end{bmatrix}\begin{bmatrix}x \\y \\1\end{bmatrix}}.}$

The parameters of the transformation model are estimated using aniterative process as follows. Least squares estimates of the parametersvalues are computed using the data set of matched features. Thetransformation model with the estimated parameter values is then appliedto the features of the captured image and the errors between thelocations of the transformed features of the captured image and thelocations of the matching features of the pattern image are evaluated.If all the errors are below an error threshold, the estimated parametersvalues are accepted as the final parameter values. If there are anyerrors above the threshold, matched features with high error are removedfrom the data set and the parameters are estimated again with thereduced set of features. The process is repeated until a set ofparameter values is estimated in which of the errors are below the errorthreshold.

After the parameter values are determined 204, the transformation modelis applied 206 to the captured image using the determined parametervalues to better align the captured image with the pattern image. Thecaptured image may then be used for depth map computation.

FIG. 3 is an example illustrating the efficacy of using the misalignmentcorrection method. This example shows a captured image with nomisalignment correction applied and the corresponding depth map and thesame captured image after misalignment correction is applied and thecorresponding depth map.

Other Embodiments

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.

For example, one of ordinary skill in the art will understandembodiments in which the misalignment correction method is executed foreach captured image.

In another example, one of ordinary skill in the art will understandembodiments in which the misalignment correction method is executed asneeded to adjust the model parameters. The new model parameters may thenbe stored and used to correct misalignment in captured images until theneed for another adjustment is determined.

In another example, one of ordinary skill in the art will understandembodiments in which the misalignment correction method is executed atregular time intervals during operation of the structured light device.At the end of each interval, the model parameters may be recomputed,stored, and used to correct misalignment in captured images during thenext interval.

In another example, one of ordinary skill in the art will understandembodiments in which the misalignment correction method is executedresponsive to a user command. The resulting model parameters may bestored and used to correct misalignment in captured images until thenext time the method is executed.

In another example, one of ordinary skill in the art will understandembodiments in which the structured light device is configured toproject multiple patterns. In such embodiments, a single suitablepattern may be used as the pattern image in the misalignment correctionmethod and the resulting parameters used to correct misalignment inimages captured using the other patterns. This single pattern may beprojected at regular intervals to estimate alignment correctionparameters or can be initiated by a user-activated control sequence whena need for adjustment is observed.

In another example, one of ordinary skill in the art will understandembodiments in which the predetermined features of the pattern imageused for matching to features of the captured image are a subset of thetotal number of features extracted from the pattern image.

In another example, one of ordinary skill in the art will understandembodiments in which the projector component is a diffraction opticalelement (DOE) that constantly projects a single fixed pattern.

Embodiments of the method described herein may be implemented inhardware, software, firmware, or any combination thereof. If completelyor partially implemented in software, the software may be executed inone or more processors, such as a microprocessor, application specificintegrated circuit (ASIC), field programmable gate array (FPGA), ordigital signal processor (DSP). The software instructions may beinitially stored in a computer-readable medium and loaded and executedin the processor. In some cases, the software instructions may also besold in a computer program product, which includes the computer-readablemedium and packaging materials for the computer-readable medium. In somecases, the software instructions may be distributed via removablecomputer readable media, via a transmission path from computer readablemedia on another digital system, etc. Examples of computer-readablemedia include non-writable storage media such as read-only memorydevices, writable storage media such as disks, flash memory, memory, ora combination thereof.

It is therefore contemplated that the appended claims will cover anysuch modifications of the embodiments as fall within the true scope ofthe invention.

What is claimed is:
 1. A method of misalignment correction in astructured light device, the method comprising: extracting features froma first captured image of a scene, wherein the first captured image iscaptured by an imaging sensor component of the structured light device,and wherein the first captured image comprises a pattern projected intothe scene by a projector component of the structured light device;matching the features of the first captured image to predeterminedfeatures of a pattern image corresponding to the projected pattern togenerate a dataset of matching features; determining values of alignmentcorrection parameters of an image alignment transformation model usingthe dataset of matching features; and applying the image alignmenttransformation model to a second captured image using the determinedalignment correction parameter values.
 2. The method of claim 1, whereinthe first captured image and the second captured image are a samecaptured image.
 3. The method of claim 1, wherein the image alignmenttransformation model is an affine transformation model modified toremove a component corresponding to a separation baseline of the imagingsensor component and the projector component.
 4. The method of claim 1,wherein the image alignment transformation model is a perspectivetransformation model modified to remove a component corresponding to aseparation baseline of the imaging sensor component and the projectorcomponent.
 5. The method of claim 1, wherein the predetermined featuresof the pattern image are a subset of a total number of features of thepattern image.
 6. The method of claim 1, wherein the extracting,matching, and determining are performed at intervals during operation ofthe structured light device.
 7. The method of claim 1, wherein thepattern is a two-dimensional high frequency binary pattern.
 8. Astructured light device comprising: an imaging sensor componentconfigured to capture images of a scene; a projector componentconfigured to project a pattern into the scene; a memory configured tostore predetermined features of a pattern image corresponding to thepattern; means for extracting features from a first image of the scenecaptured by the imaging sensor component, wherein the first imagecomprises the pattern projected into the scene by the projectorcomponent; means for matching the features of the first image to thepredetermined features to generate a dataset of matching features; meansfor determining values of alignment correction parameters of an imagealignment transformation model using the dataset of matching features;and means for applying the image alignment transformation model with thedetermined alignment correction parameter values to a second image ofthe scene captured by the imaging sensor component.
 9. The structuredlight device of claim 8, wherein the first image and the second imageare a same image.
 10. The structured light device of claim 8, whereinthe image alignment transformation model is an affine transformationmodel modified to remove a component corresponding to a separationbaseline of the imaging sensor component and the projector component.11. The structured light device of claim 8, wherein the image alignmenttransformation model is a perspective transformation model modified toremove a component corresponding to a separation baseline of the imagingsensor component and the projector component.
 12. The structured lightdevice of claim 8, wherein the predetermined features of the patternimage are a subset of a total number of features of the pattern image.13. The structured light device of claim 8, wherein the extracting,matching, and determining are performed at intervals during operation ofthe structured light device.
 14. The structured light device of claim 8,wherein the pattern is a two-dimensional high frequency binary pattern.15. A non-transitory computer-readable medium storing instructions that,when executed by at least one processor in a structured light device,cause a method of misalignment correction to be performed, the methodcomprising: extracting features from a first captured image of a scene,wherein the first captured image is captured by an imaging sensorcomponent of the structured light device, and wherein the first capturedimage comprises a pattern projected into the scene by a projectorcomponent of the structured light device; matching the features of thefirst captured image to predetermined features of a pattern imagecorresponding to the projected pattern to generate a dataset of matchingfeatures; determining values of alignment correction parameters of animage alignment transformation model using the dataset of matchingfeatures; and applying the image alignment transformation model to asecond captured image using the determined alignment correctionparameter values.
 16. The computer-readable medium of claim 15, whereinthe first captured image and the second captured image are a samecaptured image.
 17. The computer-readable medium of claim 15, whereinthe image alignment transformation model is an affine transformationmodel modified to remove a component corresponding to a separationbaseline of the imaging sensor component and the projector component.18. The computer-readable medium of claim 15, wherein the imagealignment transformation model is a perspective transformation modelmodified to remove a component corresponding to a separation baseline ofthe imaging sensor component and the projector component.
 19. Thecomputer-readable medium of claim 15, wherein the predetermined featuresof the pattern image are a subset of a total number of features of thepattern image.
 20. The computer-readable medium of claim 15, wherein theextracting, matching, and determining are performed at intervals duringoperation of the structured light device.